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Abstract. For the past few decades, instructional materials enriched with multimedia 
elements have enjoyed increasing popularity. Multimedia-based instruction incorporating 
stimulating visuals, authentic audios, and interactive animated graphs of different 
kinds all provide additional and valuable opportunities for students to learn beyond 
what conventional instruction relying mainly on print material can afford and achieve. 
Cognitive load theory, Sweller, Van Merrienboer, & Paas (1998) and Mayer’s (2001) 
theory of multimedia learning, have suggested that replacing visual text with spoken 
text (i.e., modality effect) result in better learning outcomes and that less mental efforts 
would be required to learn the multimedia lessons. The aim of this study was to test 
the generalizability of the modality effect in animation-based multimedia instruction 
developed for learning English-for-specific purposes (ESP) in an EFL classroom. 
Specifically, the study investigated the effect of spoken (audio) annotation and text 
annotation embedded in a concurrent on-screen text accompanying computer-generated 
animations that depicts the process of blood flow in a human heart. The study also looked 
into the impact of language of the annotation (i.e., students’ LI vs. L2) and interactive 
effect of language proficiency and prior knowledge and above independent variables on 
the learning outcomes and cognitive load. Results indicated no significant differences 
between LI and L2 glosses for all tests. Additionally, no significant differences were 
also found between audio and text annotations in all tests. However, L2 annotation 
significantly added more difficulties to the comprehension of the annotations than LI. 
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1. Introduction 

The use of computers as well as technology-related applications have significantly 

affected every aspect of human life, including how we receive and deliver 
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knowledge. Nowadays, people do not rely purely on print material as their major 
source of readings. Readings on the Internet, from PDAs or other electronic devices 
make a reading experience more enjoyable and move beyond a traditional linear 
fashion. Hypermedia links provided in selected nodes lead the readers to other nodes 
or links that provide extra information on the original nodes (Ariew & Ercetin, 
2004). This “nodes and links” feature of hypertext provides potentials for designing 
more flexible and richer access to conventional linear reading material for which 
a predetermined fashion of viewing is adopted. Information accessed via nodes or 
links can be used as supplementary material to further understand a topic or could 
act as an aid to further explain a difficult concept. 

With the advent of the technology information era, media of various types have 
been employed to assist learning of different kinds. However, several decades 
of research on multimedia facilitated or assisted learning have not been able to 
conclude that multimedia application is any better than conventional instruction 
without the aid of multimedia. Nevertheless, the technology era has arrived and 
deeply intertwined into every aspect of human living, it is not an issue to adopt or 
not to adopt certain technology but how to effectively adopt technology so that it 
would benefit human beings in every area. 

Annotations have been used for several decades to assist reading comprehension. 
Annotations could occur as marginal notes or at the end of reading selection for 
readers to refer to when they experience difficulties in the reading process (Ariew 
& Ercetin, 2004; Hullen, 1989). Annotations could be developed in multiple forms 
such as text (verbal), audio or visual or a combination of these three. Also, in ESL/ 
EFL texts, annotations could be provided in either students’ target language or 
mother tongue. 

Research on annotations in ESL/EFL for general English learning purposes has 
generally concluded that annotations of different types have promoted students’ 
comprehension of the reading materials or at least are equally effective as when 
none are provided. Under what conditions, though, would annotations of different 
types promote English reading for specific purposes has rarely been discussed in the 
literature. Issues like when do multimedia material embedded with different forms 
of facilitation hinder learners’ process of professional readings are not conclusively 
settled. 

In addition, do learners with individual differences in their language ability 
or prior knowledge associated with the subject matter receive different types of 
multimedia annotations equally or differently? Do different forms of annotations 
induce cognitive load differently? The purpose of the study was to compare the effect 
of audio and textual annotations in the form of either LI or L2 on EFL students’ 
learning from professional reading and induced cognitive load. Additionally the 
study explored the effect of individual difference, i.e., language proficiency and 
prior knowledge of the subject matter on above dependent variables. 
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2. Method 

2.1. Participants 

The participants were 100 undergraduate students (78 females and 22 males) who were 
enrolled in a college of humanities at a teacher-development university in Taiwan. None 
of the students were native speakers of English and had received at least 7-10 years 
of formal English education. Students’ ages were between 19 and 25. The students 
participated in the study voluntarily but were individually awarded a NT$400 book 
coupon after the study for their participation. Participants in this study were considered 
to be intermediate achievers and had passed an intermediate level of General English 
Proficiency Test (GEPT), an official general English ability test developed by The 
Language Training and Research Centre in Taiwan. 

2.2. Materials and apparatus 

The materials consisted of a multimedia animation program developed using Microsoft 
Flash, it presented a multimedia text illustrating general physiology knowledge on the 
functions of the human heart developed by Dwyer (1972). The reading text consisted 
of 2,000 words in English, presented in 20 pages. The 20 pages were further divided 
into 5 topics. Each topic consisted of approximately 200-300 words. The topic and page 
number were presented on the right side of the screen. Students were able to view their 
progress in the reading from the page numbers. 

3. Discussion 

Drawing on Mayer’s (2001) split-attention effect, which is consistent with a dual¬ 
processing model of working memory, the study tested the hypothesis that annotations 
in the form of audio would be more effective than visual (textual) in enhancing 
vocabulary learning and reading comprehension of ESP material. Additionally, the 
study explored the amount of cognitive load experienced by learners when interacting 
with different types of annotations. Interactive effects of level of annotation language/ 
medium and learners’ language proficiency and background knowledge of the ESP 
material were also diagnosed to figure out if they were additional factors that might 
co-explain the effect of annotations on students’ learning of ESP materials. The major 
findings can be summarized as following. First, students learned equally well from ESP 
material with annotations that used either their native or target language or delivered 
in the form of text or audio. Second, language proficiency was an important factor that 
would influence students’ learning from ESP material supported by annotations. High 
language proficiency students performed consistently and significantly better than their 
counterparts both in vocabulary acquisition and reading comprehension. Third, although 
types of annotations did not differentiate effects on learning outcome, different types 
of annotations did result in various amounts of cognitive loads. Annotations developed 


174 


Effects of Multimedia Vocabulary Amiotations on Vocabulary Learning and Text Comprehension... 


using students’ target language (i.e., English in this study) significantly added more 
difficulties to the comprehension of the annotations than annotations using students’ 
native language. Fourth, annotations in audio form resulted in more cognitive load 
associated with the difficulties of the annotations than visual (textual) form. Fifth, low 
proficiency students experienced higher cognitive load associated with difficulties 
of the annotations than high proficiency students. Sixth, students consistently have a 
preference for annotations provided in their native language and in visual (textual) 
form, which they also found to be more useful. 

4. Conclusions 

The results of the study did not show any significant differences in vocabulary recall 
and text comprehension between the annotations developed using students’ native 
language (Chinese) and their target language (English). This result is consistent with 
Yoshii’s (2006) study, Jacobs, Dufon, and Flong’s (1994) study, and Chen’s (2002) 
study. Flowever, previous studies conducted on whether glosses should be designed 
using LI or L2 have produced mixed results. According to Bell and LeBlanc (2000) 
and Flayden (1997), students relied on LI glosses more than L2 glosses because the 
former are more efficient in resolving their immediate lexical needs when reading a 
foreign text. When students are given the choice between L1 and L2 glosses, L2 glosses 
are seldom used (Bell & Leblanc, 2000; Davis & Lyman-Flager, 1997; Goyette, 1995; 
Hayden, 1997). 

The self-reported data using a questionnaire in this study reconfirmed the finding 
that students like LI glosses more than L2 and that LI glosses are more useful than L2 
in assisting with the comprehension, although the results in tests found no significant 
differences. Although amiotations in the format of LI did not add in more learning 
gains, annotations developed using L2 significantly added more difficulties to the 
comprehension than amiotations using LI. This finding is not surprising because L2 
annotation needs further interpretation and must become comprehensible to the learners 
before they can be used to assist vocabulary recall or text comprehension. The encoding 
process is actually conducive in potential extraneous cognitive load, but not germane 
load because it has not resulted in greater learning gains and performance of the learners. 
Extraneous cognitive load is the extra load resulting from poor instructional design. The 
level of extraneous cognitive load is determined by the format and manner in which 
the instructional material is presented and by the amount of capacity that working 
memory is used when learners engage in the instructional activities (Sha & Kaufman, 
2005). It is suspicious that annotations in students’ L2 have actually induced extraneous 
cognitive load by distracting students from performing irrelevant information coding or 
hypothesis-testing, i.e., meaning-making. 

This suspicion was further supported by the finding that cognitive load associated 
with annotations did not correlate significantly with any level of learning in this study. 
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The finding of no significant differences between LI and L2 annotations in facilitating 
vocabulary acquisition and reading comprehension has extended our understanding of 
equal or non-differential roles of language annotations played in conventional ESL/ 
EFL reading to ESP reading. Furthermore, the same finding was further extended 
to annotations that developed to assist text comprehension which was contextually 
enriched through animation. 
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